Omnimodal messaging system

ABSTRACT

A messaging server application system handles message creation, receipt, and response for any sort of digitizable message in any format. It typically operates as a core application and application infrastructure in a communications network or networks in the multiple sender, receiver and user modes of the same or varying design and operational characteristics. The system uses an application architecture organized using a set of ten loosely-coupled subsystems falling into three general functional groups: interface/connector subsystems, core subsystems, and storage subsystems. The interface/connector subsystems include voice, data and multimedia gateways as well as message connectors and a multimodal authenticator. The core subsystems include a multimedia messaging bus, an independent metadata messaging bus, and a content transformer. The storage subsystem can include various storage subsystems. The core subsystems extract message metadata and combine this metadata with information about the delivery and routing of the message to form a “Metamessage”.

BACKGROUND OF THE INVENTION

This invention relates in general to communications systems. Morespecifically, it relates to a generic, multi-media, multi-channel,messaging (server application) system. The messaging system is formed ofloosely coupled subsystems that can handle message creation, receipt andresponse for any digitalizable message, in any format, via any popularmessaging device, interface, or mode, that is received and delivered viaany popular channel.

Communications systems that interconnect users to transmit content aretypically designed using an architecture, protocols, interfaces,end-user devices, etc., that are most effective for a given type ofcontent, in a given format, transmitted on a known channel operatingwith a known protocol. A system for analog voice communications is oftenquite different from one for digital, data communications.

Ideally, a communication system is able to efficiently transmit messagesof varying content and format using any of the common communicationsdevices, channels, interfaces and modes. However, the variety of theseoptions, and the competing design considerations inherent in the choicesamong these options, has been an obstacle to the implementation of ageneric or “universal” messaging system that can receive, process anddeliver messages regardless of variations in these various system andmessage parameters. Past approaches to messaging systems have used thecommonalities among the formats of the messages they process to designthe most effective architecture for handling those formats. However,where commonalities are few, and where differences are significant,known approaches have cost or performance limitations. Some examples ofattempts to provide a more versatile messaging system are provided bygranted and applied-for patents.

Published International Patent Application WO 01/09770, for example,describes a flexible rule-based information distribution system thatselectively delivers messages generated by one or more originators toone or more recipients. The originator specifies rules to target arecipient, and a recipient also specifies rules by which to receiveinformation. The rules refer to information that appears in userprofiles and message data. There is no transformation of content, nosynchronization of multi-modal, multi-channel communications, and nogeneric user interfaces to publish to multiple node interfaces.

U.S. Pat. No. 6,345,288 discloses a communication system that transfersdata, metadata and methods from a provider computer to a consumercomputer via a network. Metadata is used to provide combined controlbetween the consumer and provider computers of the types and content ofinformation transferred. There is no separation of the metadata andmessage content, nor multi-modal synchronization of synchronous andasynchronous end user messages over multiple channels, as required in atruly universal communications system.

U.S. Pat. No. 6,401,132 describes a content transformer operating on aninput stream. There is no teaching of using such a transformer toprovide a generic, multi-mode, multi-channel, synchronous andasynchronous communications over a network.

U.S. Pat. No. 5,859,898 describes a Nynex system that can transmit bothvoice and data messages. However, it does not operate to acceptmulti-channel communication to the same device, there is no multi-modalsynchronization, no integration to external messaging systems, thesystem is not massively distributed, and its operation does not separateand process independently the metadata and content of a message.

Published US Patent Application No. US2002/0078151 discloses a systemfor communicating messages of various formats between diversecommunications devices. It can transmit over PSTN or IP networks.However, it uses a layered design that builds on these networks, not anapplication architecture operable within a network or networks. It alsooperates on messages and their metadata in the traditional manner, as anintegral message unit.

In sum, while many solutions have been proposed or used for enhancingthe capabilities of message communications systems, they solve someproblems and add some capabilities, but there is no system that is trulyuniversal and generic.

It is therefore a principal object of this invention to provide anOmnimodal messaging system that can transmit multi-media messagesreceived and delivered over multi-channels.

Another principal object is to provide a Omnimodal messaging system withthe foregoing advantages that also operates in conjunction with anycommon communications device, interface, or mode.

Still another object is to provide these advantages for communicationsdevices and protocols that are synchronous and asynchronous.

A further object is to provide the foregoing advantages in a system thatis also scalable.

Yet another object is to provide the foregoing advantages whileproviding platform independent connections to external systems.

Another object is to provide a communications system that can receive,process and deliver message content in any media as long as there is alogical mapping between the content types and a transformation moduleexists.

SUMMARY OF THE INVENTION

The present invention viewed as a multimedia messaging system connectssender and recipient end-users. The system operates in a network ornetworks each having multiple sending and recipient nodes that caninterface with other external messaging systems, messaging devices, orhuman operators. The messaging system includes three functional groups:interface/connector systems, core subsystems, and storage subsystems.They, in turn, are formed from ten subsystems.

The interface/connector subsystems receive, process and deliver messagesthat include metadata and whose message content that can differ frommessage to message and type and be delivered to and from devices andcomputer platforms that can differ in type, over delivery channels,protocol and interface.

The core subsystems including a content transformer, a multimediamessaging bus, and a metadata messaging bus, said buses interacting withall the interface/connector functional subsystems and the contenttransformer. The interface/connector, storage and core subsystemsoperate to:

-   -   i. asynchronously, and with adequate processing resources,        synchronously, produce, process and deliver a Metamessage on the        meta data messaging bus independent of the processing and        delivery of the original message, with the Metamessage        containing instructions for processing the original message and        the Metamessage used by the rest of the system to determine how        it will process the original message;    -   ii. reformat and translate all message content, if necessary, to        a form compatible with the recipient; and    -   iii. synchronize the receipt, processing and delivery of the        messages to deliver such multimedia messages and to synchronize        messages with other external messaging systems.

Using this system the end-users are enabled to manage their messages inan Omnimodal and in an asynchronous and synchronous manner includingmessages of different content types and formats in support of differentinterfaces, messaging devices, and modes of creation, receipt andresponse, via plural channels, using plural protocols, in support ofplural messaging platforms. The core subsystems are loosely coupled—theyinteract, but they operate independently so that changes in onecomponent does not necessarily require changes in other interconnectedcomponents.

Viewed as a process, the present invention is a method forinterconnecting plural communication devices communicating multimediamessages in a network or networks via sending and recipient nodes withmessaging devices and systems or human operators external to the networkor networks. It includes as steps:

-   -   interfacing with said networks, devices, systems and operators,        and then separating the message being communicated into        multimedia message output (or content) and related metada. A        central aspect of this invention is the step of creating,        processing and delivering a Metamessage on a bus independently        of the multimedia message being delivered and processed.    -   This processing includes reformatting and translating and        storing message content as needed to be compatible with a        recipient node. Another step involves synchronizing the receipt,        processing, and delivery of the multimedia message content with        external messaging systems.    -   The invention will be more fully understood from the following        detailed description which should be read in light of the        accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the overall messaging serverapplication architecture of the present invention;

FIG. 2. is a more detailed block diagram of the voice user interfacegateway shown in FIG. 1;

FIG. 3 is a more detailed block diagram of the data gateway shown inFIG. 1 illustrated to show pure HTTP and/or Web Service connections;

FIG. 4 is more detailed block diagram of the messaging connector shownin FIG. 1; and,

FIG. 5 is a more detailed block diagram of the content transformer shownin FIG. 1.

DETAILED DESCRIPTION OF THE INVENTION

The messaging server application system 10 according to the presentinvention, shown in overview in FIG. 1, is designed to handle messagecreation, receipt, and response for any sort of digitizable message inany format, including, but not limited to, voicemail, email, short text(Short Message Service (SMS)), Multimedia Messaging Service (MMS),instant messages, and faxes, via any popular end user messaging device(phone, mobile phone, handheld computer, desktop/laptop computer, faxmachine, converged devices), via any popular interface (WirelessApplication Protocol (WAP) browser, voice interface, WAP/voice, SMSclient, MMS client, Java client, Brew (Binary Run-time Environment forWireless) client, web browser, thick IM client, etc.), in any mode(text, audio, still image, moving images, or combination thereof), andreceived or delivered via any popular channel (Public Switch TelephoneNetwork (PSTN), Internet, etc.). It is thus multi-modal. The messageshandled by the system 10 can vary in content among the messages, orwithin the same message. The messaging server system 10 with thesecharacteristics is termed, herein, “Omnimodal”. It typically operates asa “core application and application infrastructure” in a communicationsnetwork or networks in the multiple sender, receiver and user modes ofthe same or varying design and operational characteristics. Themessaging system 10 is assembled through machine-to-machine and/orhuman-to-machine interfaces.

This generic or universal messaging system 10, termed herein as“Omnimodal”, uses a multi-media messaging server applicationarchitecture organized using a set of ten loosely-coupled subsystems:These subsystems, as detailed below, fall into three general functionalgroups 11, 19, and 26.

Interface/Connector Subsystems 11

1. Voice User Interface Gateway 12

2. Data Gateway 14

3. Multimedia Gateway 16

4. Message Connectors 18

5. Multimodal Authenticator 15

Core Subsystems 19

6. Multimedia Messaging Bus 20

7. Metadata Messaging Bus 22

8. Content Transformer 24

Storage Subsystems 26

9. Storage Subsystems (various) 26

The first five of these subsystems 12-18 are interface/connectorsubsystems. They all interact with the world external to theapplication. They support all the interfaces. They also manageconnections to external telecommunications and data networks as well asto external messaging systems. They are responsible for sending andreceiving any popular kind of message in any popular mode for anypopular device, as detailed above.

The next three subsystems 20, 22, 24 can be thought of as the brains orcore of the system. They extract message metadata (data about messages),including message type, format, mode of creation, address, originatingdevice, subscriber, etc. They combine this metadata with informationabout the delivery and routing of the message provided by the networkinginfrastructure, information encapsulated in the user preferences and theuser registry, as well as with instructions on how to process themessage and the Metamessage itself. All these elements are containedwithin an element termed the “Metamessage” (Metamessage is“reflective”). The Metamessage is processed to determine what the systemmust do to deliver the original messaging; what content transformations(if any) need to be performed on the original message; what formats andinterfaces will be used to deliver the original message. Original ortransformed parts of the original message and/or a forerunner messagemay then be sent to external facing subsystems that then handledelivery.

The last set of subsystems, the Storage Subsystems 26, store all of theinformation used by the system, namely the messages themselves,Metamessages, subscriber preferences, and registry data.

The system 10 handles any format, and avoids any architecturalcommitments that rely on format commonalties. The resulting system canbe termed “format independent.” The core subsystems reduce any messageto two sets of data—the message, and data about the message. The onlyassumption relied upon by the system is that all messages can be reducedto binary data. The Content Transformer 24 includes algorithms forconverting message formats.

The loosely-coupled nature of the subsystems enables modifications toone subsystem to occur without necessitating modifications to theothers. As times goes on and new message formats are introduced into themarket, this system will readily accommodate these new formats. Anadditional layer need not be added. To handle the new format, the system10 simply adds a connector or interface to the interface/connectorsubsystems 12-18, adds format conversion capability to the ContentTransformer 24, and adds any relevant compression technology to thestorage subsystem 26. The architecture itself need not change. “Looselycoupled” as used herein means that while the subsystems are operativelyinterconnected, they operate generally independently. For example, thecontent transformer operates asynchronously on message content aspresented via the buses 20 and 22. Also, a Metamessage is created anddelivered on the bus 22 independently of the associated multi-mediamessage content carried on the bus 20. In the preferred form, the buses20 and 22 are software buses, not hard wire buses, or the like.

Subsystem Descriptions

Voice Gateway 12

As shown in FIG. 2, this subsystem 12 enables reception of any type ofvoice message through voice-specific channels 30 such as PSTN, IPtelephony, packet-switched telephony, and other cellular telephony ofwhich three representative channels are illustrated. The Voice Gateway12 is an entry point that accommodates all of the different methods ofconnecting to the voice user interface, and makes all voice relatedinteractions with the system appear the same to the rest of the system.The Voice Gateway 12 enables the other components of the system to treatvoice without the concern for how the voice was obtained, or what formatit is in. The Voice Gateway 12 itself is rendered withstandards-compliant VoiceXML, thereby adhering to the Extensible MarkupLanguage (XML) mandate of the architecture. The Voice Gateway 12includes functionality that enables it to synchronize multi-partmessages that have one or more types of content. Since the Voice Gateway12 is only designed to handle the voice portion of the content, throughany voice channel, it will use a synchronization mechanism 32(SMIL—Synchronized Multimedia Integration Language—compliant) to workwith the Data Gateway 14, the Multimedia Messaging Bus 20, and othercomponents in the system 10.

Data Gateway 14

Much of the interaction between outside systems 34 (shown in FIG. 3 asthree representative such systems 34) and the system 10 is done throughthe Data Gateway 14. The Data Gateway 14 also handles connections forsubscribers. The specific subscriber interfaces are rendered, by proxy,through the Content Transformer 24 and then sent out through the DataGateway 14. The Data Gateway enables reception of any type of datamessage through data specific channels 36 such as HTTP/XML, W-HTTP,i-mode, and BREW. The data gateway 14 offers access to the system 10through various web service types including Simple Object AccessProtocol (SOAP) 36 a and Xforms 36 b. SOAP 36 a allows externalapplications to communicate with the data gateway independent of thecomputer platform. SOAP 36 a can be thought of as an XML schema forremote procedure calls, like message retrieval. XForms enables renderingof generic interfaces. XForms interfaces can be transformed to specificuser interface types at the node. In this way carriers can renderproprietary subscriber interfaces that have pre-built interaction setswith the application. As a result, mobile carriers will be able tocreate their own subscriber interfaces in a more flexible and lesscostly way by simply creating one set of Extensible Stylesheet LanguageTransformations (XSLTs). Whereas a simple XML interface requires thecustomer to create workflow code, XForms streamlines this process byeliminating that requirement. XForms 36 b is a World Wide Web Consortium(W3C) standard. By providing these two distinct types of web serviceinterfaces (SOAP and XForms), the application optimizes connectivityoptions for users of the system 10. The end result is an applicationthat is more flexible and less costly to deploy than anything nowavailable.

Full Multimedia Gateway 16

The subsystem 16 serves the same general function as the Data Gateway14, but is designed to receive/send any type of multimedia file ormessage format such as MMS, Moving Picture Experts Group (MPEG), MPEG-4,MPEG-7, FLIC, Audio Visual Interleaved (AVI), QuickTime Movie (MOV),Artificially Structured Films (ASF), Macromedia Flash, etc.

Message Connectors 18

The subsystem 18 shown in FIG. 4 is designed to connect with externalmessaging systems such as Simple Mail Transfer Protocol (SMTP), PostOffice Protocol (POP), Internet Messaging Access Protocol (IMAP), ShortMessage Service Center (SMSC), and Multimedia Messaging Service Center(MMSC). This subsystem is able to exchange messages with other messagingsystems outside of the system 10, as well as with the MultimediaMessaging Bus 20 and Metadata Messaging Bus 22.

Multimodal Authenticator 15

The major task of the Multimodal Authenticator is to provide a securemechanism for end users, be it user devices or foreign networks, tocommunicate with the Voice Genesis system. The Multimodal Authenticatorprovides authentication, authorization, and secure communication betweenall external interfaces to the Voice Genesis server systems and allforeign systems. This subsystem provides the following functionality:(1) Authenticating the user through a variety of modalities(voice-print, text-based password, etc.) In this way, we provide auser-friendly mechanism for authentication and authorization regardlessof modality. (2) Integration with foreign systems that provide variouslevels of authorization or group-wise authentication. (3) A mechanism bywhich the distribution of users' messages amongst many differentlocations is made possible. The Multimodal Authenticator is used tofacilitate an intelligent distribution of the data based on the lastknown connection status of the end user device. (4) Integration withcarrier authentication systems. This is done so that carriers canprovide services using the system 10 along side other services and theauthentication is seamless to the end user. For example, if a useralready has a user-name and password in the carrier network, theMultimodal Authenticator enables the Voice Genesis servers to accept thesame set of credentials for authentication and authorization.

In implementation, the Multimodal Authenticator is built by integratingexisting standard authentication, authorization, and encryptionmechanisms such as private-public key exchanges used in TLS/SSL. Incases where not relevant, such as voice printing (recognizing the userby using a pre-recorded voice segment that confirms liveliness andauthentication), suitable known technique to modality is used toauthenticate and authorize the user in a secure manner. While theMultimodal Authenticator is an amalgamation of existing authenticationtechniques and tools, it connects these tools and techniques together tooffer a seamless and user-friendly authentication and authorizationexperience to the end users as well as facilitate ease of integrationwith foreign systems, each of which may have their own authenticationand authorization systems.

Multimedia Messaging Bus 20

The Multimedia Messaging Bus 20 allows different types of media to beput in a queue and then processed. It solves several differentrequirements. First, it allows coordinated access to all of the contentsof the message (regardless of what type of content is inside themessage) by different processing subsystems (Content Transformer 24,Storage Subsystem 26, etc.). Second, it provides this access in ascalable and asynchronous manner. As a result spikes in message trafficdo not cause the system 10 to halt. Finally, it permits the content ofthe message to be retrieved at run-time while the information about themessage (on the Metadata Messaging Bus 22) is replicated to all of thedifferent nodes on a distributed network.

Metadata Messaging Bus 22

The Metadata Messaging Bus 22 transports Metamessages (as defined above)between subsystems, so subsystems can coordinate to process messages. Inorder to provide a decoupling between messages and information about themessages in the presently preferred form of the invention, theInterface/Connector Subsystems 11 create Metamessages that contain dataabout the original messages. These Metamessages are then placed on theMetadata Messaging Bus 22. The Metamessages are themselves provided asmessages on a queue. This enables the clients of the Multimedia MessageBus 22 to know needed information about the messages prior to actuallyprocessing the messages. This approach provides tunable performance andscalability.

Content Transformer 24

The subsystem 24 shown in detail in FIG. 5 transforms any popularmessage format into any other popular message format to facilitatemessage creation and delivery via any popular format, interface, device,mode, or channel. In one form of the invention that is more easilyimplemented, content transformation is done asynchronously, that is, ona deferred basis. This approach lessens the performance and scalabilityrequirements inherent to synchronous systems. Transformation appears tobe synchronous to the extent that processing resources are available. Inother words, while the invention in its present preferred form operatesasynchronously, that asynchronous operation can appear to besynchronous, e.g., if there is no queuing on the buses 20 and 22.

Transformable content includes any combination of text, still images,audio, and moving images. Because messages may include any combinationof these content modes, the total number of combinations is twenty-fouron both the sending and receiving side. These modes each containmultiple formats that must be supported.

Storage Subsystems 26

The set of subsystems 26 handles storage of the various content piecesof stored messages. The messages and parts of messages, whether text,still images, audio, and/or moving images, must be stored. Thesesubsystems are comprised of several off the shelf components amongstwhich the most important are:

Text Message Storage: The database 26 a is used to store the messagemetadata to facilitate searches, queries, and data mining. The database26 a may also be used to store the text portion of messages.

File Storage: The multimedia or voice files are stored in their nativeformat in a file storage systems 26 c and 26 b, respectively, and thenbe processed by the Content Transformer 24 to be played back to theuser. The Content Transformer 24 can also write back to the StorageSubsystems 26 to use them as a caching mechanism, or to providedifferent types of file formats. There are a variety of file formatseach of which may require a particular type of storage as the systemscales. Several standard compression methods are used to facilitatestorage of various formats. The Storage Subsystems 26 a-c as shown arealso designed to support streaming delivery of messages to recipients.

LDAP: A Lightweight Directory Access Protocol (LDAP) implementation isused with storage subsystem 26 to find the location of the stored files.LDAP is a set of protocols based on standards within the X.500 standard,but simplified, and allows any type of Internet access. It runs almostany application and is compatible with all popular computer platforms.Java Naming and Directory Interface (JNDI) interfaces are provided tofacilitate fail-over capabilities.

The subsystems 26 can be considered as a single storage subsystem withsub-subsystems associated with various message types, and one or moresub-subsystem for message management and retrieval.

While this invention has been described with respect to its preferredembodiments, it will be understood that various modifications andalterations will occur to those skilled in the art from the foregoingdetailed description and the accompanying drawings. For example, othernumbers of subsystems can be used intercommunicating through other busarchitectures besides two parallel buses that open multi-media messagesand Metamessages. The system 10 is readily scalable, up or down. In eachvariation, however, data about a message is separated from the messageitself; a new message is then created to hold this data; the originalmessage and new message are processed as independent messages; the newmessage is used by the system to determine how to process the originalmessage. While it is possible to utilize non-bus network topologies,e.g. a star or ring topology, the bus implementation described andillustrated is presently preferred as the most readily implementedtopology that touches all of the subsystems. As used herein, the term“bus” is intended to include these alternative topologies. Stillfurther, the two buses 20 and 22 can be implemented as a single bususing known transmission management techniques such as interleaving.

1-2. (canceled)
 3. A method for interconnecting plural communicationdevices communicating multimedia messages and metadata in a network ornetworks via sending and recipient nodes with messaging devices andsystems or human operators external to the network or networks,comprising: interfacing with said networks, devices, systems andoperators, separating the message being communicated into multimediamessage output and related metadata, creating, processing and deliveringa metamessage on a bus independently of said multimedia message beingdelivered and processed, said processing including reformatting andtranslating and storing message content as needed to be compatible witha recipient node, and synchronizing said receipt, processing, anddelivery of the multimedia message content and the external messagingsystems.